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ABSTRACT 


In  this  paper  we  describe  a  color  image  segmentation  system  that  performs  color 
clustering  in  a  color  space  followed  by  color  region  segmentation  in  the  image  domain. 
In  color  space,  we  describe  two  different  algorithms  that  cluster  similar  colors  using 
different  criteria  and  present  our  evaluation  results  on  these  two  algorithms  in  comparison 
with  three  well-known  color  segmentation  algorithms.  The  region  segmentation 
algorithm  merges  clusters  in  the  image  domain  based  on  color  similarity  and  spatial 
adjacency.  We  developed  three  different  methods  for  merging  regions  in  the  image 
domain.  The  color  image  segmentation  system  has  been  implemented  and  tested  on  a 
variety  of  color  images,  including  satellite  images  and  moving  car  images.  The  system 
has  shown  to  be  both  effective  and  efficient. 
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Color  Image  Segmentation  in  the  Color  and  Spatial  Domains 
1.  Introduction 

In  the  past  decade,  color  imaging  has  become  popular  in  many  applications  including 
object  classification  and  recognition,  video  surveillance,  image  indexing  and  retrieval  in 
image  databases  and  video  databases,  feature  based  video  compression,  etc.  [1,  2].  In 
some  applications,  image  contents  can  be  better  described  in  terms  of  color  features 
combined  with  spatial  relations  such  as  enclosure  and  adjacency  due  to  the  irregularity  of 
object  shapes  in  images.  This  paper  describes  our  research  in  color  image  segmentation, 
which  is  often  a  necessary  computational  process  for  color-based  image  retrieval  and 
object  recognition  [3]. 

Image  segmentation  is  a  process  of  partitioning  image  pixels  based  on  selected  image 
features.  The  pixels  that  belong  to  the  same  region  must  be  spatially  connected  and  have 
similar  image  features.  If  the  selected  segmentation  feature  is  color,  an  image 
segmentation  process  would  separate  pixels  that  have  distinct  color  features  into  different 
regions  and,  simultaneously,  group  pixels  that  are  spatially  connected  and  have  similar 
color  into  the  same  region.  In  color  imagery,  image  pixels  can  be  represented  in  a 
number  of  different  color  spaces,  such  as  RGB,  XYZ,  or  LUV  [4,  5,  6].  One  major 
concern  in  color  image  segmentation  is  that  the  computational  complexity  increases 
significantly  in  comparison  with  gray  scale  image  segmentation.  The  process  of^in^^ 
segmentation  can  be  considered  as  unsupervised  clustering,  if  a  priori  knowledge  about 
the  number  and  type  of  regions  present  in  the  image  is  not  available  [7,  8].  Image 
clustering  procedures  often  use  the  individual  image  pixels  as  units  and  compare  each 
pixel  value  with  every  other  neighboring  pixel  value,  which  requires  excessively  long 
computation  times  for  images  of  high  resolutions.  A  high  quality  representation  of  color 
requires  8  bits,  using  PCM  (Pulse  Code  Modulation)  quantization  for  each  of  the  three 
color-components.  Red  (R),  Green  (G)  and  Blue  (B).  For  each  image  pixel,  24  bits  of 
amplitude  quantization  is  required,  resulting  in  2^“  =  16,777,216  distinguishable  colors. 
This  leads  to  high  computational  cost  in  image  clustering. 

In  this  paper  we  describe  an  efficient  color  image  segmentation  system,  depicted  in 
Figure  1.  The  color  image  segmentation  system  consists  of  two  stages  of  computation. 
At  the  first  stage,  we  use  a  color  clustering  algorithm  to  generate  clusters  of  similar  colors 
in  the  color  histogram  space  of  an  image.  The  histogram  of  a  color  image  in  a  selected 
color  space  is  a  three  dimensional(3D)  discrete  feature  space  that  provides  the  color 
distribution  of  the  image.  The  output  of  the  color  clustering  algorithm  is  a  set  of  non¬ 
overlapping  color  clusters,  CL,.  Each  cluster  in  CL,  contains  similar  colors  and  all  colors 
in  the  same  cluster  are  assigned  with  the  same  color  label.  The  labeling  of  the  clusters  in 
CL,  results  in  a  multi-threshold  image  in  which  pixels  of  the  same  region  have  the  same 
color  label  and  are  spatially  connected.  These  regions  form  the  second  Cluster  set  CLj  to 
be  used  in  the  second  stage,  where  a  region  segmentation  algorithm  agglomerates  the 
initial  clusters  in  CLj,  based  on  the  spatial  connection  and  the  color  distances  between  the 
adjacent  regions.  The  second  stage  merges  the  selected  adjacent  regions  and  does  riot 
split  any  regions.  Therefore,  the  first  design  criterion  for  the  fuzzy  clustering  algorithm  is 
that  different  color  regions  should  be  in  different  clusters  in  CL,.  However,  we  should 
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prevent  the  generation  of  too  large  a  set  of  clusters.  For  example,  one  extreme  is  that  each 
cluster  in  CL,  contains  only  one  color,  which  certainly  satisfies  the  first  criterion  but 
misses  the  purpose  of  the  histogram  based  clustering.  Therefore,  the  second  design 
criterion  is  that  CL,  must  be  compact.  The.se  two  criteria  have  been  u.sed  to  guide  the 
development  of  the  clustering  algorithms. 

The  second  set  of  clusters,  CL,,  is  obtained  by  labeling  image  pixels  with  the 
corresponding  color  clusters  in  CL,.  Because  there  is  no  spatial  information  used  at  the 
first  stage,  the  pixels  in  the  same  cluster  in  CL,  can  be  scattered  over  the  entire  image, 
resulting,  in  many  cases,  ICLJ  »  ICL,l.  The  segmentation  algorithm  at  the  second  stage 
iteratively  merges  the  regions  in  CLj  based  on  the  color  distances  between  the 
neighboring  regions,  the  region  sizes  and  the  maximum  number  of  clusters  in  CL,,  the 
system  output  that  contains  the  meaningful  color  regions  with  respect  to  objects  in  the 
image. 


color  image 


Figure  1.  An  overview  of  the  color  image  segmentation  system. 


This  paper  is  organized  as  follows.  Section  2  will  describe  two  different  algorithms  that 
perform  the  clustering  computation  in  color  space  and  their  performance  evaluation. 
Section  3  will  present  a  region  segmentation  algorithm  that  groups  clusters  in  the  image 
domain  based  on  color  similarity  and  spatial  adjacency.  Section  4  presents  our 
experimental  results  and  the  performance  analysis  of  each  algorithm  in  the  segmentation 
system. 

2.  Clustering  in  color  space 

Color  is  the  most  popular  feature  used  in  image  retrieval  and  object  recognition 
applications  for  color  imagery.  Color  is  often  described  in  terms  of  intensity,  luminance, 
lightness,  hue  and  saturation.  Intensity  is  a  measure,  over  some  interval  of  the 
electromagnetic  spectrum,  of  the  flow  of  power  that  is  radiated  from,  or  incident  upon,  a 
surface.  Intensity  is  what  is  called  a  linear-light  measure,  expressed  in  units  such  as  watts 
per  square  meter.  There  are  a  number  of  color  spaces  that  are  defined  based  on  different 
application  criteria  and  most  of  these  are  three-dimertsional  [3].  In  general,  for  a  given 
color  space,  we  can  calculate  a  color  histogram  function  F(C)  from  a  color  image  /, 
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where  C.  is  a  color  in  the  color  space  and  F(C.)  denotes  the  number  of  pixels  in  I  that  have 
color  C.  The  dimensions  of  a  color  histogram  are  determined  by  the  color  space  used  to 
represent  the  image.  For  example,  in  the  L*u*v*  space,  Cj  is  represented  by  a  vector  (1, 
u,  v)  and  F(C,J  is  the  number  of  pixels  that  have  L*u*v*  values  equal  to  (l,u,v);  in  RGB 
color  space,  C;  is  represented  by  a  vector  (r,  g,  b)  and  F(C.)  is  the  number  of  pixels  in  / 
that  have  color  equal  to  (r,  g,  b).  Considering  the  uncertain  nature  of  classifying  similar 
colors  into  clusters,  we  developed  two  color-clustering  algorithms,  Max  Quantization 
Error,  and  Optimal-Cut. 


Max  Quantization  Error 

This  is  a  greedy  algorithm  that  aggressively  reduces  the  global  color  quantization  error  by 
creating  new  clusters  at  the  place  where  the  maximum  quantization  error  is.  This 
algorithm  has  the  following  steps: 

1.  CU={)  and  j=0. 


2.  Find  the  color  bin  in  the  histogram  with  the  maximum  color  frequency,  and  create 
a  color  cluster  centered  in  that  bin.  Mathematically,  we  are  trying  to  find  a  color, 

such  that 

^(^ctacr)  -  for  all  color  Q  in  the  image  domain. 
q='c,„,^^.andCLI=CL]^{CJ 

3.  Calculate  color  quantization  error  for  every  bin  in  the  histogram  using  the 
following  equation: 

£,=F(C,|C,-C,f 


4.  Find  such  that 


-C 

cluster  ^  j 


^clusler  ~  cluster  )|l^i 

set  j=j-t-l,  Cj  =  ,  and  CL1=CL1  kj{CJ 


>  El  for  all  i. 


5. 


Recalculate  the  color  quantization  eiTor  for  every  color  C,  in  the  histogram  using 
the  following  equation: 

,  ~  ~  ^nearest _ctusier  ’ 

where  e  CL\  is  the  cluster  that  is  closest  to  C,. 


6.  Find  the  color,  with  the  maximum  color  quantization  error  defined  in  step  4, 

set  j=j+ 1 ,  ,  and  CL/ = CL7  u /C.y 

7.  Repeat  steps  5  to  6  until  one  of  the  following  conditions  is  met  then  go  to  step  8: 

a.  ICLll  =  M,  where  M  is  the  desired  number  of  clusters  set  by  the  user 

b.  the  maximum  quantization  error,  is  below  a  threshold. 


8.  At  the  end  the  loop  we  have  CLI={  C\i=0,  1,  M-1},  which  is  a  set  of  colors 

representing  M  cluster  centers.  For  every  color  C  in  the  image,  C  is  assigned  a 
color  label  p,  0<p<M-l,  if 

0.*(CC,)  =  lC-C,f  <D„,./(C.C,)=||C-C,|f, 
for  all  j  such  that  0<j<  M-1. 

Optimal-Cut 

The  optimal-cut  algorithm  adopts  a  splitting  process  that  generates  a  number  of  color 
cubes  in  a  color  space  that  minimize  the  global  color  quantization  error.  The  algorithm  is 
based  on  the  searching  of  3D  boxes,  i.e.  a  right  rectangular  prisms,  in  the  color  space  that 
give  minimized  global  color  quantization  error. 

We  define  the  quantization  error  function  within  a  3D  box  in  the  color  space  as 
follows: 

Eh, a  =  11e(c,}\c^-cJ 

iehox 

where  c.  is  the  3-dimensional  ith  color  vector  within  box,  F(c)  is  the  color  frequency  (or 
the  number  of  pixels  that  have  the  color)  of  c.,  and  is  the  central  color  of  the  box, 

which  is  calculated  as  follows: 

— .  CjShox 

Cj^hox 

The  optimal-cut  algorithm  has  the  following  steps 

1.  Define  a  minimum  3D  box,  i.e.  a  right  rectangular  prism,  box,,  =(0,,  Q,  C,,  Q,  C,, 
Cf,,  C,,  C^)  in  the  color  space  that  encompasses  all  the  colors  in  the  image  I,  where 
C.,  i  =  1,  2,  ...,  8  are  the  3D  coordinates  of  the  eight  vertices  that  uniquely  define 
the  color  box.  Add  it  to  the  cluster  list  CLl  =  {  box,, } . 

2.  For  each  color  box  in  CLl,  search  along  each  of  the  three  coordinate  axes  in  the 
color  space  to  find  a  split  point  that  can  partition  the  box  into  two  sub-boxes  and 
where  the  reduction  of  the  quantization  error  is  maximum  in  comparison  to  all  the 
other  possible  splits  within  all  current  color  boxes.  Assume  a  color  box  has 
coordinates  box={(x„  y„  z,),  (x^,  y,,  z,),  (x„  y„  z^),  (x^,  y,,  z^),  (x„  y^,  z,),  (x^,  y„ 
z,),(  X,,  yj,  Zj),  (Xj,  yj,  z^)}.  If  a  split  occurred  along  the  x-axis  at  point  x*,  where 
X,  <  X*  <  Xj  ,  then  the  two  sub-boxes  have  coordinates,  sub-box l={(x,,  y,,  z,), 
(Xj,  y,,  z,),  (x,,  y,,  z,),  (x*,  y,,  z^),  (x„  y„  z,),  (x*,  y,,  z,),(  x,,  y^,  zj,  (x*,  y^,  z^)}, 
and  sub-box2  =  {(x*,  y„  z,),  (x^,  y,,  z,),  (x*,  y,,  z^),  (x^,  y„  z^),  (x*,  y^,  z,),  (x^,  y^, 
z,),  ( x*,  y^,  Zj),  (Xj,  yj,  Zj)}.  Splits  along  the  y-axis  and  the  z-axis  can  be  similarly 
calculated.  The  reduction  of  quantization  error  resulted  from  the  split  is 
calculated  as  follows: 

Eftox  Eg^h-hoxl  Eg„f,-hnxl  ' 


Based  on  the  definition  of  the  quantization  error  given  above,  it  can  be  easily  shown 
that  "t" 

3.  Readjust  the  two  sub-boxes  so  that  there  are  no  empty  color  bins  along  their 
boundaries  and  then  add  them  to  the  color  cluster  set  CLl. 

4.  Repeat  steps  2  and  3  until  the  desired  number  of  color  clusters  is  obtained,  or  the 
maximum  quantization  error  is  below  the  threshold. 

We  compared  the  two  color  clustering  algorithms  introduced  above  with  three  other 
algorithms  published  in  the  literature.  Pair-wise  clustering.  Medium-cut  and  Octree  [9, 
10,  1 1].  Figure  2  shows  the  experiment  results  on  three  different  images  generated  by  the 
proposed  two  methods  as  well  as  the  other  three.  All  algorithms  were  set  to  generate  1 6 
different  color  clusters.  From  the  resulting  images  we  can  see  that  the  proposed  optimal- 
cut  and  max-quantization  generated  better  results  than  the  other  three  algorithms  on  all 
three  example  images. 


Medium-cut  Octree 


(a)  Color  clustering  experiments  conducted  on  image  1 . 


Medium-cut  Octree 

(b)  Color  clustering  experiments  conducted  on  image  2. 
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(c)  Color  clustering  experiments  conducted  on  image  3. 

Figure  2  Experimental  evaluation  of  various  color-clustering  algorithms. 


We  further  evaluate  the  five  algorithms  by  measuring  the  quantization  errors  on  the 
resulting  images.  Table  1  shows  the  quantization  error  generated  by  the  five  color 
clustering  algorithms  on  the  three  images  shown  in  Figure  2. 


Table  1:  Evaluation  ol 

F  quantization  errors  of  five  co 

lor  clustering  a 

Igorithms 

optimal -cut 

Medium-cut 

Octree 

Pair-wise 

14.9 

15.0 

26.2 

36.4 

37.9 

31.1 

15.8 

31.8 

37.5 

46.4 

15.5 

15.4 

33.9 

32.9 

44.0 

Both  the  Max  Quantization  Error  and  Optimal -cut  algorithms  gave  much  better 
performance  in  terms  of  minimizing  the  quantization  error  on  all  three  images  than  the 
other  three  algorithms.  Both  algorithms  gave  similar  performance  in  reducing  the  global 
color  quantization  error  for  image  1  and  image  3.  However  the  Max  Quantization  Error 
algorithm  performed  much  better  on  image  2  than  Optimal-cut. 

According  to  [9]  the  Pair-wise  clustering  algorithm  should  produce  less  quantization  error 
than  other  algorithms  such  as  uniform,  popularity,  median-cut,  local  K-means,  variance- 
based,  and  octree.  However,  the  Pair-wise  algorithm  needs  to  maintain  a  quantization 
error  matrix  that  can  be  overwhelmingly  large,  which  makes  this  algorithm  impractical. 
For  example,  for  a  24-bit  color  image,  every  color  component  has  256  levels.  If  we  use 
32  levels  to  generate  the  histogram  (as  most  algorithms  do),  we  will  have  32^  =  32,768 
bins.  Therefore,  initially,  the  quantization  error  matrix  can  have  32,768^  =  1,073,741,824 
elements.  In  these  experiments,  when  we  used  clustering  level  16,  the  Pair-wise 
clustering  algorithm  took  significantly  longer  than  the  other  four  algorithms  and  it  did  not 
give  better  performance. 


3.  Region  segmentation  in  image  domain 

Region  segmentation  is  implemented  by  a  spatial  clustering  algorithm  that  groups  the 
clusters  generated  by  the  color  clustering  algorithms  mentioned  in  the  previous  section 
using  various  measurements. 

When  we  map  the  color  clusters  in  CL,  to  the  image  domain,  we  obtain  a  color  cluster  set 
CL^,  in  which  each  cluster  contains  image  pixels  that  are  both  spatially  connected  and 
within  the  same  color  cluster  in  CL,.  In  general,  one  cluster  in  CL,  can  be  decomposed 
into  more  than  one  cluster  in  CL^,  and  therefore  CL,  is  much  larger  than  CL,.  This  is 
evidenced  by  the  color  clustered  images  shown  in  Figure  2.  Pixels  of  the  same  color  can 
be  scattered  all  over  the  image  domain,  which  results  in  many  different  clusters  in  CL^.  In 
general,  ICL,! « ICL^I. 

The  image  segmentation  algorithm  in  the  image  domain  is  an  agglomerative  process  that 
uses  the  following  three  parameters: 

•  color  distances  among  neighboring  clusters  in  the  spatial  domain, 

•  cluster  sizes  and 

•  maximum  number  of  clusters  in  CL,. 

After  every  merge  of  two  clusters,  the  center  of  the  new  cluster  is  calculated,  and  the  size 
and  the  neighbors  of  the  merged  cluster  are  updated.  We  define  the  distance  between  two 
clusters  as  the  color  distance  between  the  centers  of  the  two  clusters.  During  the 
sequential  merging  process,  an  important  issue  is  the  order  of  merging  clusters,  which  can 
significantly  affect  the  region  segmentation  result.  We  have  investigated  three  clustering 
merging  methods.  All  of  the  three  methods  use  a  common  parameter,  maxjcls,  to  control 
the  maximum  number  of  clusters  in  CL,.  The  three  methods  differ  in  the  priority  of 
selecting  clusters  to  merge  at  each  iterative  step  and  will  depend  on  the  requirements  of 
various  applications. 

Method  1  attempts  to  merge  the  adjacent  clusters  that  are  similar  in  colors.  In 
implementation,  we  use  a  control  parameter,  cl_diffjh  to  denote  the  color  difference 
threshold.  At  the  first  step,  the  algorithm  attempts  to  merge  the  neighboring  clusters 
whose  color  distances  are  below  cl_diff_th.  The  order  of  merging  is  not  considered  at  the 
this  step.  If  the  number  of  clusters  at  the  end  of  the  first  step  is  greater  than  max_cls,  then 
the  algorithm  begins  the  second  step.  At  the  second  step,  the  algorithm  selects  the 
smallest  cluster  and  merges  the  cluster  with  one  of  its  neighbors  to  which  it  has  the 
smallest  color  distance.  This  merging  process  repeats  until  the  number  of  clusters  in  CL, 
is  no  more  than  mcix_cls. 

Method  2  considers  the  size  of  clusters  as  the  only  selection  criterion.  It  selects  the 
smallest  cluster  and  merges  the  cluster  with  one  of  its  neighbors  to  which  it  has  the 
smallest  color  distance.  The  process  is  repeated  until  the  number  of  clusters  in  CL,  is  no 
more  than  max_c/.y. 

Method  3  considers  the  color  distance  as  the  most  important  criterion  in  cluster  merging 
in  the  image  domain.  However,  the  computation  required  in  finding  the  minimum  color 


distance  between  two  adjacent  clusters  is  quite  time  consuming  if  the  number  of  clusters 
in  CL^  is  large.  To  alleviate  the  computational  burden,  the  algorithm  consists  of  three 
passes  of  merging.  At  the  first  pass,  it  repeatedly  merges  the  smallest  clusters  with  their 
neighbors  that  have  the  closest  color  distance  until  the  total  number  of  clusters  is  reduced 
to  a  reasonable  number.  At  the  second  pass,  the  algorithm  selects  a  pair  of  two  adjacent 
clusters  that  has  the  smallest  color  distance  within  the  entire  image  to  merge.  This 
process  repeats  until  the  top  max_cls  clusters  in  size  contain  a  large  percentage  of  the 
image  pixels.  Since  the  largest  max_cl  clusters  already  cover  the  majority  of  the  entire 
image  pixels,  the  small  clusters  below  the  top  max_cl  should  not  affect  too  much  the  final 
segmentation  result.  Therefore  at  the  third  pass,  the  algorithm  repeatedly  merges  the 
smallest  cluster  with  its  closest  neighbor  in  color  distance  until  the  total  number  of 
clusters  in  CL,  is  no  more  than  max_cls. 

From  a  computational  point  of  view.  Methods  1  and  2  are  more  efficient  than  Method  3. 
However,  in  many  cases.  Method  3  generates  better  results  than  the  other  two  methods. 


4.  Implementation,  experiments  and  conclusion 

All  the  algorithms  described  in  this  paper  have  been  implemented  in  C++  under  the 
Windows  2000®  operating  system.  In  the  implementation,  we  first  converted  color  from 
RGB  coordinates  to  L*u*v*. 

We  have  tested  the  segmentation  system  on  a  large  number  of  images  from  video 
sequences  taken  from  a  moving  vehicle.  Due  to  the  limited  space,  we  use  one  image 
example  to  illustrate  the  performance  of  the  proposed  clustering  system.  Figure  3(a) 
shows  an  image  from  a  video  image  sequence  taken  in  a  city  street  scene  by  a  video 
camera  mounted  on  a  moving  vehicle.  The  objects  of  interest  in  this  application  are 
vehicles  in  front  of  the  primary  vehicle,  which  has  the  video  camera  mounted.  In  these 
experiments,  we  used  both  Max  Quantization  Error  and  Optimal-Cut  in  the  color  space 
clustering  and  Method  3  in  region  segmentation.  It  appears  that  Optimal-Cut  gave  better 
segmentation  results  in  the  color  space.  The  results  from  region  segmentation  that  follow 
both  Max  Quantization  Error  and  Optimal-Cut  separated  the  vehicles  from  their 
surrounding  objects  such  as  road,  trees,  etc.  However,  the  Optimal-Cut  appeared  to  give 
better  details. 


(b)  Color  clusters  generated  by  Optimal  '  Color  clusters  generated  by  Max 
cut  Quantization  Error 


(d)  Image  segmentation  generated  by  Optimal-cut  (e)  Image  segmentation  generated  by  Max 
with  50  clusters.  Quantization  error  with  50  clusters. 


Figure  3  Image  segmentation  using  optimal-cut  and  max 
quantization  error. 


Figure  4  shows  the  region  segmentation  results  when  different  number  of  clusters  were 
specified.  In  comparison  to  the  results  shown  in  Figure  3,  where  cluster  number  50  was 
used,  we  can  conclude  that  when  the  number  of  clusters  increases,  more  details  are  shown 
but  more  region  fragments  may  result.  In  general  Optimal-Cut  followed  by  region 

segmentation  gives  effective  object  segmentation  using  color  features. 
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Figure  4  Image  segmentation  using  different  number  of  clusters  in  image  domain. 
Max  Quantization  Error  followed  by  region  segmentation  with  (a)  cluster  number 
100  and  (b)  cluster  number  25.  Optimal-Cut  followed  by  region  segmentation  with 
(c)  cluster  number  100  and  (d)  cluster  number  25. 


Unlike  many  existing  clustering  algorithms,  the  image  segmentation  system  does  not 
require  the  knowledge  about  the  number  of  the  color  clusters  to  be  generated  at  each  stage 
and  the  resolution  of  the  color  regions  can  be  controlled  by  one  single  parameter,  the 
radius  of  a  cluster.  The  color  image  segmentation  system  has  been  implemented  and 
tested  on  a  variety  of  color  images  including  satellite  images,  car  and  face  images.  The 
system  has  shown  to  be  both  effective  and  efficient. 
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